1. Field of the Invention
The present invention relates to a method and apparatus for tracking moving objects in real time whereby trajectories corresponding to the movement of the objects are determined. More particularly, the present invention relates to a method and apparatus for tracking moving objects, such as athletes, in connection with sporting events and exploiting such tracking to derive information corresponding to the movement of the objects being tracked.
2. Description of Related Art
Tracking systems for moving objects typically generate trajectories corresponding to the motion of an object within the view of a camera. The trajectories or tracks typically consist of a sequence of x, y (location) coordinates and time coordinates. The information from these trajectories has a variety of applications. For example, the information can be used to count the number of objects, such as a people or vehicles, crossing a reference line and to associate a particular direction with each crossing. In addition, such trajectories may be used to determine the number of people present within the field of view of a camera at any instant, which information is useful, for example, for product marketing such as determining the effectiveness of a particular advertisement or advertising technique in a store. Tracking systems may also be employed for measuring consumer traffic throughout, for example, the aisles of a store, etc., including the length of time that particular persons spend in specific aisles.
One field in which real time tracking would be particularly desirable, but is not currently greatly utilized, is in the field of sports.
For example, most sports activities are measured in terms of statistics which highlight various aspects of the sport, such as personal performance (number of points scored, fastest time, etc.), comparative performance (e.g., number of points scored versus opponent), and playing strategy (e.g., number of perimeter baskets scored in a basketball game versus number of inside baskets scored).
Conventionally, most such statistics can be readily measured (e.g., number of points scored, time for running a race, number of errors committed). However, other statistics are of interest, but are relatively difficult to obtain readily, and are certainly difficult to obtain in connection with real time tracking of an event. Such statistics may include distance traveled by a player per an interval of time (e.g., a game, set, quarter, etc.), instantaneous and average player speed and acceleration, shot selection (e.g., in hockey or basketball), or areas of a playing field (court, rink, etc.) covered by a player.
Generally, real time tracking of athletes in a sporting event is challenging, especially because it is difficult to obtain a clean segmentation of an athlete from the background in view of changing lighting conditions, variations in clothing worn by athletes (especially with regard to color), differences in the visual characteristics (for example, reflectivity) of playing surfaces (grass, clay, hardwood, ice, etc.), and the fast and dynamic movement of athletes. Another factor complicating tracking is the simple presence of other moving objects or moving people, in addition to the athlete being tracked.
Several methods or systems have been developed for the tracking of moving objects, including people. However, these conventional systems do not yield a single motion region or even a consistent set of motion regions, which deficiencies are exacerbated when tracking athletes in the midst of highly dynamic movement.
For example, in Rashid, R. F., xe2x80x9cTowards A System For The Interpretation Of Moving Light Displaysxe2x80x9d, 2 IEEE Transactions on Pattern Analysis and Machine Intelligence, 574-581 (1980), a method is described for interpreting moving light displays (MLD). In general, Rashid teaches segmenting out from MLD images individual points corresponding to moving people. The individual points are grouped together to form clusters based on, inter alia, the positions and velocities of the individual points; the formed clusters represented individual objects. Tracking is performed by matching points between consecutive frames based on the relative distances between the location of points in the current frame and the location of predicted points in a previous frame. The predicted position is based on the average velocity of the point in the previous frame and the relative distance, which is calculated using a Euclidean function.
The technique described by Rashid has several drawbacks. Specifically, the MLD system requires several frames before a good object separation is obtained, and no criteria is provided for determining when satisfactory object separation has occurred. In addition, no mechanism is provided for propagating the generated clusters to prior and subsequent frames for continuity in the motion representation. This undermines real time operation.
In another tracking system described in Rossi, M. and Bozzoli, A., xe2x80x9cTracking And Counting Moving Peoplexe2x80x9d, Proceedings Of The Second IEEE International Conference On Image Processing, 212-16(1994), a vertically mounted camera is employed for tracking and counting moving people. This system operates under the assumption that people enter a scene along either the top or bottom of the image where altering zones are positioned for detecting people moving into the scene. In reality, however, people can also appear in a scene, inter alia, from behind another object or from behind an already-identified person. In other words, people may be wholly or partially occluded upon initially entering a scene and would not be identified by this system. The problem of identifying occluded persons is also present in the system described in Rohr, K., xe2x80x9cTowards Model Based Recognition Of Human Movements In Image Sequencesxe2x80x9d, 59 Computer Vision, Graphics And Image Processing: Image Understanding, 94-115(1994). Such problems are clearly pertinent to real time tracking of athletes during a sporting event.
In addition, the systems described in Smith, S. M., and Brady, J. M., xe2x80x9cA Scene Segmenter: Visual Tracking of Moving Vehiclesxe2x80x9d, 7 Engineering Applications Of Artificial Intelligence 191-204(1994); and xe2x80x9cASSET-2: Real-Time Motion Segmentation And Shape Trackingxe2x80x9d, 17 IEEE Transactions On Pattern Analysis And Machine Intelligence, 814-20 (1995), are designed specifically for tracking objects such as moving vehicles, and accordingly identify features representing corners or abrupt changes on the boundaries of the vehicles. This approach is based on that tracking objects which are rigid with unchanging contours, and, thus, permits the use of constant velocity or constant acceleration models, techniques clearly unavailable for tracking of people, particularly when tracking people in motion, particularly the dynamic motion of sports where hands and feet may all move with different speeds and in different directions.
In the related U.S. Pat. No. 5,764,283, an apparatus and method are disclosed for tracking moving objects in real time. In particular, an apparatus and method are disclosed in which local features, such as extrema of curvature on boundary contours, are tracked, and trajectories of motion are derived by dynamically clustering the paths of motion of the local features.
The present invention provides, most generally, a method and apparatus for tracking moving objects, particularly athletes engaged in sporting activities. More particularly, the invention provides an apparatus and method for obtaining information corresponding to the athlete(s) and/or sporting activity being tracked. In one embodiment of the present invention, such information is performance statistics for an athlete derived from real time tracking of that athlete. In another embodiment, such information is an occupancy map, which may be visually manifested, corresponding to the frequency with which an athlete occupies particular regions of a playing field, or the time spent in particular regions of the playing field. In yet another embodiment of the present invention, such information is embodied in a computer-generated replay of the sporting event (or some part thereof) using computer generated characters moving in accordance with motion data collected from real time tracking of the sporting event. In particular, the present invention is intended to be especially useful for enhancing television coverage of sporting events.
The method according to the present invention, in pertinent part, includes the steps of matching and merging a select feature path (representing the motion of the one of usually a plurality of local features of an athlete) with a candidate cluster of feature paths chosen from a plurality of preexisting clusters representing motion of the athlete or object. The select feature path has a portion concurrent or overlapping in time with some of the clusters in the plurality of clusters. The candidate cluster is preferably chosen as the cluster closest in distance to the select path. Once the candidate cluster is chosen and the select feature path merged therewith, the parameters of the candidate cluster are updated based on the parameters of the select path.
The distance between the select path and the preexisting clusters is calculated in accordance with a function having two components. The first component is representative of a maximum displacement between the select feature path and the candidate cluster, and the second component is representative of the difference between velocity vectors defining motion of the select feature path and velocity vectors defining motion of the candidate cluster.
Finally, the method includes steps of using the clusters extrapolated in order to provide real time information about the athlete or the sporting event being tracked. For example, a trajectory of an athlete derived according a cluster of feature paths extrapolated in accordance with the foregoing may be mapped onto a video image in order to provide a visual illustration of the athlete""s path of movement. From this, for example, an occupancy map may be generated illustrating the frequency or time spent by the athlete in certain portions of a playing field. Also, the trajectory of the athlete can be analyzed with respect to time in order to provide performance information such as instantaneous and average speed and acceleration and the like. In another example, the trajectory of the athlete can be used to generate computer-driven replays of an athletic event, whereby activity on the playing field is duplicated based on player trajectories derived from real time video tracking.
An apparatus constructed in accordance with the present invention particularly includes a device for calculating the distance between a select feature path representing motion of the features of an object and a plurality of preexisting clusters, some of which overlap in time with the select feature path. The distance is calculated using a function having a first component and a second component. The first component is representative of a maximum displacement between the select feature path and each preexisting cluster, and the second component is representative of the difference between velocity vectors defining motion of the select feature path and velocity vectors defining motion of each preexisting cluster. Another device is also provided for selecting, from the preexisting clusters, a candidate cluster having the smallest distance to the select feature path. In a preferred embodiment, the functions of the distance calculating device and selecting device are both embodied in a computer.
Also, in a preferred embodiment, the apparatus includes a device for obtaining information corresponding to the athlete(s) or sporting event being tracked, such as performance statistics for respective athletes, and a device for compiling and storing the performance statistics. The device for calculating performance statistics may be also embodied in the same computer carrying out the functions of the distance calculating device and the selecting device discussed above.
Other objects and features of the present invention will become apparent from the following detailed description considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims.